Telepresence based inventory pick and place operations through robotic arms affixed to each row of a shelf

ABSTRACT

Disclosed are a system and/or a method of telepresence based inventory pick and place operations through actuator controlled robotic arms affixed to each row of a shelf. A method includes mounting a robotic arm at an end of a row of a shelf of inventory on a set of rails affixed to the row of a shelf. The robotic arm is permitted to move horizontally along the row of the shelf. The robotic arm is repositioned along the three axes using a set of actuators. A haptic motion of a human user is mirrored that is remotely using a positioning device (e.g., human may feel the feedback of the remote arm as it touches the objects). An item is placed on a counting platform in front of the robotic arm. The items are placed automatically in the designated location down through a transport means when a pick operation is completed.

CLAIM OF PRIORITY

This application is the Continuation application of and claims priorityto, and incorporates herein by reference the entire specification of theU.S. Utility patent application Ser. No. 14/726,458 titled ‘TELEPRESENCEBASED INVENTORY PICK AND PLACE OPERATIONS THROUGH ROBOTIC ARMS AFFIXEDTO EACH ROW OF A SHELF’ filed on May 30, 2015. This U.S. Utilityapplication Ser. No. 14/726,458 further claims priority to the U.S.Provisional Application No. 62/145,488 titled TELEPRESENCE BASEDINVENTORY PICK AND PLACE OPERATIONS THROUGH PNEUMATIC ROBOTIC ARMSAFFIXED TO EACH ROW OF A SHELF filed on Apr. 9, 2015.

FIELD OF TECHNOLOGY

This disclosure relates generally to robotics, and more particularly toa method, apparatus, and system of telepresence based inventory pick andplace operations through robotic arms affixed to each row of a shelf.

BACKGROUND

A distribution center (e.g., a warehouse, a fulfillment center, across-dock facility, a bulk break center, an unstructured or structuredstorage area, and/or a package handling center) may be a physical spacein which inventory (e.g., products) is temporarily stored and/or handledfor downstream delivery to retailers or consumers. The distributioncenter may allow a single location to stock a number of products.Distribution centers may be located far from urban locations to securelower costs or closer to where customers are located to facilitate sameday and/or next day delivery (e.g., for e-commerce transactions), andmay be in urban locations where costs of real estate per square foot isexpensive. For this reason, the distribution center may be stacked highin a small space, with rows reaching ten (10) meters or more. Activitieswithin manufacturing facilities may cover—picking of parts for assemblyfrom a vertically stacked storage area with many high rows of shelving.In addition, some organizations may operate manufacturing and/ordirect-to-consumer distribution in a single facility or interconnectedfacility to share investments (e.g., in space, equipment, laborresources, and/or inventory as applicable).

The distribution center may have a series of rows having stackedshelving. Items may be stored on these shelves. A warehouse managementsystem (e.g., WMS system) may be used to identify and track inventory inthe distribution center. A human lift may be required to pick itemsstored in a higher row of the shelf based on information from thewarehouse management system. This may require repositioning of warehousevehicles. In addition, a trained human operator may need to enter a liftbasket and manually perform tasks such as picking items from highershelves. This may result in significant labor expenses in training andskilled labor. Further, professional hazard insurances for workplaceinjury and occupational risks may be high as these tasks may bedangerous and accident prone. Therefore, the distribution center mayoperate inefficiently and may be expensive and/or hazardous to operate.

SUMMARY

Disclosed are a system and/or a method of telepresence based inventorypick and place operations through robotic arms affixed to each row of ashelf. In one aspect, a method includes mounting a robotic arm at an endof a row of a shelf of inventory on a set of rails affixed to the row ofa shelf. The method further includes permitting the robotic arm to movehorizontally along the three axes along the row of the shelf. Therobotic arm is repositioned along the three axes using a set ofactuators. The set of actuators are a backdrivable, an electrosensing,an electric, a magnetic, a hydraulic, and/or pneumatic actuators.

Further, the method includes contemporaneously mirroring a haptic motionof a human user that is remotely using a positioning device that iscommunicatively coupled with the robotic arm through an Internetnetwork. The robotic arm is automatically repositioned along the threeaxes using the set of actuators responsive to the haptic motion of thehuman user that is remotely using the positioning device.

The method includes placing an item of inventory on a counting platformin front of the robotic arm using an end effector of the robotic arm(based on an action of a human that views the item of inventory) througha camera. The camera is affixed to the robotic arm which iscommunicatively coupled with a computing device associated with thehuman user operating the positioning device. The method includes movingthe item of inventory to a designated location adjacent to the roboticarm using the end effector. The items are placed automatically in thedesignated location down through a transport means adjacent at one endof the shelf when a pick operation is completed. The designated locationis a tote and/or a storage bin. The transport means is a tube and/or alift platform to bring the tote from the designated location to adesired location of the shelf.

The method may include placing the item of inventory on a platformhaving an angled surface such that the storage bin in which the item ofinventory is placed is angled upward and does not fall off the shelfwhen placed on the counting platform. In addition, the item of inventorymay be validated based on a weight of an item on the counting platform.Also, the method may include a telepresence application to provide thepositioning device control over the robotic arm remotely through theInternet network. The shelf may include a plurality of rows of the shelfand a plurality of robotic arms. Each of the rows of the shelf mayinclude one and/or more robotic arms that horizontally traverse aparticular row in which it is affixed.

Each of the robotic arms may coordinate with a warehouse managementserver to automatically direct the human user of the telepresenceapplication to each location on the shelf associated with the itemsneeded to be fulfilled in an e-commerce order. Each item of ane-commerce order may be automatically deposited through respective onesof the plurality of robotic arms into a packing box on a conveyor beltof a distribution center. The items may be deposited down through thetube adjacent to the one end of the shelf.

The telepresence application may detect that the human has completed amovement of the items onto the counting platform and/or directs thehuman user to a next one of a plurality of the robotic arms positionedin front of the storage bin where a selection is needed.

In addition, a neural network may automatically monitor a behavior ofthe human user. The neural network may continuously learn how to improvethe pick and/or place of a particular type of item from the shelf suchthat, over time, a control program may learn how to control the roboticarm to automatically select the items onto the counting platform withouta human intervention or a human haptic control.

The end effector of the robotic arm may be a supple rubber end point, agripping arm, a sticky polymer end, an impactive end effector, aningressive end effector, an astrictive end effector and/or a contigutiveend effector. The robot may have two robotic arms with actuator controlto permit the human user to grab and/or lift the items (e.g., the systemmay operate with two parts—compliance to prevent users and/or operatorsbeing injured and detection of objects). The robotic arm may safelyoperate around the human operators in a distribution center by slowlycoming to a stop and/or not having to stop abruptly through complianceto prevent users and/or operators being injured (when an adjacent objectis detected which is likely to obstruct a motion of the robotic arm).

In another aspect, a method of a telepresence application includesdetermining that a remote positioning device associated with a humanuser is communicatively coupled with the telepresence application. Themethod also includes validating the human user as an authorized user ofthe telepresence application using a processor and a memory of awarehouse management server. A robotic arm is automatically allocated ina distribution center that is assigned to the human user based on ane-commerce order.

In addition, the method includes contemporaneously repositioning therobotic arm along the three axes using a set of actuators (based on amirrored haptic motion of the human user that is remotely using apositioning device). The method may provide a view of an item ofinventory directly in front of the robotic arm to the human user who isremotely controlling the robotic arm through the positioning device. Therobotic arm may place the item of inventory on a counting platform infront of the robotic arm using an end effector of the robotic arm basedon an action of a human that views the item of inventory through acamera.

The camera may be affixed to the robotic arm which is communicativelycoupled with a computing device associated with the human user operatingthe positioning device based on a place command from the telepresenceapplication. Further, the robotic arm may validate the item of inventory(based on a weight of the item) on the counting platform based on avalidate command from the telepresence application.

The robotic arm may move the item of inventory in a location (adjacentto the robotic arm) using the end effector based on a move command fromthe telepresence application. The method may include moving items at thelocation down through a tube adjacent to one end of the shelf when apick operation is completed based on a complete command from thetelepresence application.

A shelf may include a plurality of rows of the shelf and a plurality ofrobotic arms. Each of the rows of the shelf may include a differentrobotic arm. Each of the rows of the shelf may include one and/or moredifferent robotic arms that horizontally traverse a particular row inwhich it is affixed.

In yet another aspect, a warehouse system includes a set of robotic armsthat are each affixed to the individual rows of shelving in adistribution center to reposition horizontally along individual rows ofshelving along the three axes using a set of actuators. The warehousesystem also includes an Internet network. The warehouse system includesa warehouse management server having a telepresence application coupledwith the set of robotic arms through the Internet network. Thetelepresence application provides remote control to a human user overeach of a set of the robotic arms through the Internet network.

The system also includes a positioning device coupled with each of theset of robotic arms and the warehouse management server having thetelepresence application through the Internet network. The set ofrobotic arms is haptically controlled by a human user in a manner suchthat a haptic motion of the human user of the positioning device isimitated through a mirrored repositioning of the set of robotic arms.

The methods and systems disclosed herein may be implemented in any meansfor achieving the various aspects, and may be executed in the form of anon-transitory machine-readable medium embodying a set of instructionsthat, when executed by a machine, cause the machine to perform any ofthe operations disclosed herein.

Other features will be apparent from the accompanying drawings and fromthe detailed description that follows.

BRIEF DESCRIPTION OF DRAWINGS

The embodiments of this invention are illustrated by way of example andnot limitation in the figures of the accompanying drawings, in whichlike references indicate similar elements and in which:

FIG. 1 is a schematic view of a warehouse system illustrating a set ofrobotic arms coupled with a warehouse management server through anInternet network, according to one embodiment.

FIG. 2 is a partial view of FIG. 1 illustrating the motion of a roboticarm, according to one embodiment.

FIG. 3 is a network view illustrating the authentication and monitoringof a human user, according to one embodiment.

FIG. 4 is a conceptual view illustrating the mirroring of a hapticmotion of the human user that is using a positioning device, accordingto one embodiment.

FIG. 5 is a user interface view showing a display screen of FIG. 4,according to one embodiment.

FIG. 6 is a pick operation view, according to one embodiment.

FIG. 7 is a partial view showing dual robotic arms performing the pickand place operation, according to one embodiment.

FIG. 8 is a critical path view illustrating a flow based on time wherethe pick and place operation is performed in the warehouse system ofFIG. 1, according to one embodiment.

FIG. 9 is a process flow diagram illustrating automatic placement ofitems in a designated location when the pick operation is completed,according to one embodiment.

FIG. 10 is a process flow diagram illustrating repositioning of therobotic arm using a set of actuators based on a mirrored haptic motionof the human user, according to one embodiment.

FIG. 11 illustrates various embodiments of the robotic arm of FIG. 1,according to one embodiment.

FIG. 12 is a schematic diagram of exemplary data processing devices thatcan be used to implement the methods and systems disclosed in FIG. 1,according to one or more embodiments.

Other features of the present embodiments will be apparent from theaccompanying drawings and from the detailed description that follows.

DETAILED DESCRIPTION

Example embodiments, as described below, may be used to provide a methodand/or system of telepresence based inventory pick and place operationsthrough robotic arms affixed to each row of a shelf.

In one embodiment, a method includes mounting a robotic arm 100 at anend of a row 102 of a shelf of inventory (e.g., shelf 104) on a set ofrails 202 affixed to the row 102 of a shelf 104. The method furtherincludes permitting the robotic arm 100 to move horizontally along thethree axes along the row 102 of the shelf 104. The robotic arm 100 isrepositioned along the three axes using a set of actuators 204. The setof actuators 204 are a backdrivable, an electrosensing, an electric, amagnetic, a hydraulic, and/or pneumatic actuators.

Further, the method includes contemporaneously mirroring a haptic motion402 of a human user 304 that is remotely using a positioning device 314that is communicatively coupled with the robotic arm 100 through anInternet network 108. The robotic arm 100 is automatically repositionedalong the three axes using the set of actuators 204 responsive to thehaptic motion 402 of the human user 304 that is remotely using thepositioning device 314.

The method includes placing an item of inventory 206 on a countingplatform 208 in front of the robotic arm 100 using an end effector 210of the robotic arm 100 (based on an action of a human user 304 thatviews the item of inventory 206) through a camera (e.g., camera forremote human user 212). The camera (e.g., camera for remote human user212) is affixed to the robotic arm 100 which is communicatively coupledwith a computing device 302 associated with the human user 304 operatingthe positioning device 314 (e.g., positioning arm, data gloves). Themethod includes moving the item of inventory 206 to a designatedlocation adjacent to the robotic arm 100 using the end effector 210. Theitems 218 are placed automatically in the designated location downthrough a transport means 706 adjacent at one end of the shelf 104 whena pick operation 608 is completed. The designated location is a tote 214and/or a storage bin 106. The transport means 706 is a tube and/or alift platform 120 to bring the tote 214 from the designated location(e.g., tote 214, storage bin 106) to a desired location (e.g., on theconveyor belt 110) of the distribution center.

The method may include placing the item of inventory 206 on a platform216 having an angled surface such that a storage bin 106 in which theitem of inventory 206 is placed is angled upward and does not fall offthe shelf 104 when placed on the counting platform 208. In addition, theitem of inventory 206 may be validated using validation function 310based on a weight of an item (e.g., item of inventory 206) on thecounting platform 208. Also, the method may include a telepresenceapplication 312 to provide the positioning device 314 (e.g., positioningarm, data gloves) control over the robotic arm 100 remotely through theInternet network 108. The shelf 104 may include a plurality of rows(e.g., row 102) of the shelf 104 and a plurality of robotic arms (e.g.,robotic arm 100). Each of the rows (e.g., row 102) of the shelf 104 mayinclude one and/or more robotic arms (e.g., robotic arm 100) thathorizontally traverse a particular row (e.g., row 102) in which it isaffixed.

Each of the robotic arms (e.g., robotic arm 100) may coordinate with awarehouse management server 112 to automatically direct the human user304 of the telepresence application 312 to each location on the shelf104 associated with the items 218 needed to be fulfilled in ane-commerce order. Each item (e.g., item of inventory 206) of ane-commerce order may be automatically deposited through respective onesof the plurality of robotic arms (e.g., robotic arm 100) on a conveyorbelt 110 of a distribution center. The items 218 may be deposited downthrough the transport means 706 (e.g., a tube) adjacent to the one endof the shelf 104.

The telepresence application 312 may detect that the human (e.g., humanuser 304) has completed a movement of the items 218 onto the countingplatform 208 and/or directs the human user 304 to a next one of aplurality of the robotic arms (e.g., robotic arm 100) positioned infront of a storage bin 106 where a selection is needed.

In addition, a neural network 306 may automatically monitor a behaviorof the human user 304. The neural network 306 may continuously learn howto improve the pick and/or place of a particular type of item (e.g.,item of inventory 206) from the shelf 104 such that, over time, acontrol program may learn how to control the robotic arm 100 toautomatically select the items 218 onto the counting platform 208without a human intervention or a human haptic control.

The end effector 210 of the robotic arm 100 may be a supple rubber endpoint, a gripping arm, a sticky polymer end, an impactive end effector,an ingressive end effector, an astrictive end effector and/or acontigutive end effector. The robot 702 may have two robotic arms 704with actuator control to permit the human user 304 to grab and/or liftthe items 218 (e.g., the system may operate with two parts—compliance toprevent users and/or operators being injured, hurt, or killed anddetection of objects). The robotic arm 100 may safely operate around thehuman operators in a distribution center by slowly coming to a stopand/or not having to stop abruptly through compliance to prevent usersand/or operators being injured (when an adjacent object is detectedwhich is likely to obstruct a motion of the robotic arm 100).

In another embodiment, a method of a telepresence application 312includes determining that a remote positioning device 314 (e.g.,positioning arm, data gloves) associated with a human user 304 iscommunicatively coupled with the telepresence application 312. Themethod also includes validating (using validation function 310) thehuman user 304 as an authorized user of the telepresence application 312using a processor 114 and a memory 116 of a warehouse management server112. A robotic arm 100 is automatically allocated in a distributioncenter that is assigned to the human user 304 based on an e-commerceorder.

In addition, the method includes contemporaneously repositioning therobotic arm 100 along the three axes using a set of actuators 204 (basedon a mirrored haptic motion 402 of the human user 304 that is remotelyusing a positioning device 314). The method may provide a view (throughthe camera for remote human user 212) of an item of inventory 206directly in front of the robotic arm 100 to the human user 304 (on thedisplay screen 406) who is remotely controlling the robotic arm 100through the positioning device 314. The robotic arm 100 may place theitem of inventory 206 on a counting platform 208 in front of the roboticarm 100 using an end effector 210 of the robotic arm 100 based on anaction (haptic motion 402) of a human (e.g., human user 304) that viewsthe item of inventory 206 through a camera (e.g., camera for remotehuman user 212).

The camera (e.g., camera for remote human user 212) may be affixed tothe robotic arm 100 which is communicatively coupled to a computingdevice 302 associated with the human user 304 operating the positioningdevice 314 (e.g., positioning arm, data gloves) based on a place commandfrom the telepresence application 312.

The robotic arm 100 may move the item of inventory in a location(adjacent to the robotic arm 100) using the end effector 210 based on amove command from the telepresence application 312. The method mayinclude moving the items 218 in the location (e.g., on the conveyor belt110) down through a tube (e.g., a lift platform 120) adjacent to one endof the shelf 104 when a pick operation 608 is completed based on acomplete command from the telepresence application 312.

A shelf 104 may include a plurality of rows (e.g., row 102) of the shelf104 and a plurality of robotic arms (e.g., robotic arm 100). Each of therows (e.g., row 102) of the shelf 104 may include a different roboticarm 100. Each of the rows of the shelf 104 may include one and/or moredifferent robotic arms (e.g., robotic arm 100) that horizontallytraverse a particular row (e.g., row 102) in which it is affixed.

In yet another embodiment, a warehouse system 150 includes a set ofrobotic arms (e.g., robotic arm 100) that are each affixed to individualrows (e.g., row 102) of shelving in a distribution center to repositionhorizontally along individual rows of shelving along the three axesusing a set of actuators 204. The system also includes an Internetnetwork 108. The warehouse system includes a warehouse management server112 having a telepresence application 312 coupled with the set ofrobotic arms (e.g., robotic arm 100) through the Internet network 108.The telepresence application 312 provides remote control to a human user304 over each of a set of the robotic arms through the Internet network108.

The system also includes a positioning device 314 coupled with each ofthe set of robotic arms (e.g., robotic arm 100) and the warehousemanagement server 112 having the telepresence application 312 throughthe Internet network 108. The set of robotic arms (e.g., robotic arm100) is haptically controlled by a human user 304 in a manner such thata haptic motion 402 of the human user 304 of the positioning device 314is imitated through a mirrored repositioning of the set of robotic arms(e.g., robotic arm 100).

FIG. 1 is a schematic view of a warehouse system 150 illustrating a setof robotic arms (e.g., robotic arm 100) coupled with a warehousemanagement server 112 through an Internet network 108, according to oneembodiment. Particularly, warehouse system 150 shows a robotic arm 100,a row 102, a shelf 104, a storage bin 106, Internet network 108, aconveyor belt 110, a warehouse management server 112, a processor 114, amemory 116, database 118, and a lift platform 120, according to oneembodiment.

A robotic arm 100 may be a type of electro-mechanical arm designed toexecute pick, move, and/or place operations. A robotic arm 100 may bedesigned to be operated and/or controlled by a human user 304.Conversely, the robotic arm 100 may be programmed and then left alone torepeat their tasks independent of the control of human user 304. The row102 may be a horizontal section of the shelf of inventory (e.g., shelf104) usually used to occupy storage bin 106 filled with the items 218,according to one embodiment.

The shelf 104 may be rectangular structure with horizontal slab used inthe distribution centers to provide a surface to hold the items 218 fordisplay, storage and/or offer for an e-commerce sale. An Internetnetwork 108 may be a group of computing devices (e.g., hardware andsoftware) that are linked together through communication channels (e.g.,wired, wireless) to facilitate communication in the distributioncenters. The storage bin 106 may be an open container on the shelf 104where the items 218 can be stored, according to one embodiment.

The conveyor belt 110 may be a continuous moving surface that transportspacking box from one place to another in the distribution center. Thewarehouse management server 112 may be a computer on which the program(e.g., telepresence application 312) runs. The warehouse managementserver 112 may be capable of accepting requests from the human user 304who is operating the telepresence application 312 and respondaccordingly, according to one embodiment.

A processor 114 may be a central unit of the warehouse management server112 containing the logic circuitry to perform all the basic instructionsof a computer program (e.g., telepresence application 312). The memory116 may be an electronic holding place for instructions, information,and/or data temporarily or permanently that the processor 114 needs. Thedatabase 118 may be a collection of information that is organized sothat it can easily be accessed, managed, and/or updated. A lift platform120 attached to the one end of the shelf 104 may be an elevator thatraises and/or lowers to transport the items 218 between different levelsin the shelf 104, according to one embodiment.

FIG. 1 illustrates the set of robotic arms (e.g., robotic arm 100)communicatively coupled to the warehouse management server 112 throughan Internet network 108. The warehouse management server 112 may includea database 118 coupled with the processor 114 and memory 116, accordingto one embodiment.

FIG. 2 is a partial view 250 of FIG. 1 illustrating the motion of arobotic arm 100, according to one embodiment. Particularly, partial view250 shows a set of rails 202, a set of actuators 204, an item ofinventory 206, a counting platform 208, an end effector 210, a camerafor remote human user 212, tote 214, a platform 216, items 218, andcamera 220, according to one embodiment.

The set of rails 202 may be a horizontal bar of metal affixed across therows (e.g., row 102) of the shelf 104 supported by a framing member atthe two ends of the row 102 that serve as a guide and/or running surfacefor the robotic arm 100. The set of actuators 204 may be a mechanicaldevice that converts electrical, chemical, and/or thermal energy into amechanical energy (e.g., physical motion). The physical motion may belinear and/or rotary motion. The set of actuators 204 may be used in therobots (e.g., robot 702) for providing the power to the robot jointsand/or robotic arm 100, according to one embodiment.

The robotic arm 100 may be activated by a backdrivable, anelectrosensing, an electric, a magnetic, a hydraulic, and/or pneumaticactuators. A backdrivable actuator may be an electro-mechanical devicefor actively moving and/or driving the robotic arm 100 with high forcesensitivity and/or high impact resistance which adapts to quick externalforce mechanically. Backdrivability may be essential for safe operationof robotic arm 100 around people, operating in unstructured environments(e.g., distribution center, warehouse) for stable control of contactforces. A pneumatic actuator may be an electro-mechanical deviceactivated, controlled, and/or powered by air and/or gas pressure,according to one embodiment. In one alternate embodiment, a wide rangeof any type of compliant actuators may be utilized when activating therobotic arm 100.

An item of inventory 206 may refer to the goods being sold in ane-commerce order. A counting platform 208 may be a weighing platform tocalculate the weight of the item of inventory 206. The counting platform208 may have same level as that of the storage bin 106, according to oneembodiment. An end effector 210 may be a device and/or a tool connectedat the end of a robotic arm 100 that interacts with the work environmentand/or executes an action. The end effector 210 may be a supple rubberend point, a gripping arm, a sticky polymer end, an impactive endeffector, an ingressive end effector, an astrictive end effector, and/ora contigutive end effector, according to one embodiment.

A camera for remote human user 212 may be an optical instrument mountedat one of the joints of the robotic arm 100 that provides close-up viewof the items 218 inside the storage bin 106 to the human user 304through display screen 406. A tote 214 (e.g., designated location) maybe a container to hold an item of inventory 206 and/or items 218. A tote214 may be a tilted removable tote and/or bucket in one alternateembodiment. The platform 216 may be a raised surface which holds thestorage bin 106. The platform 216 may be angled upward so that the items218 does not fall off the shelf 104 during the pick and/or placeoperation, according to one embodiment.

The items 218 may be the collection of finished goods (e.g., item ofinventory 206) placed in the storage bin 106 for ecommerce. A camera 220may be an optical device mounted at the top of the row 102 just abovethe storage bin 106 to provide a close up view to the human user 304,according to one embodiment.

In circle ‘1’, the item of inventory 206 is placed on a countingplatform 208 using an end effector 210 based on a haptic motion 402 ofthe human user 304 that is remotely operating the positioning device 314(e.g., positioning arm, data gloves), according to one embodiment. Incircle ‘2’, the item of inventory 206 is moved into a designatedlocation (e.g., tote 214, storage bin 106) adjacent to the robotic arm100 using the end effector 210, according to one embodiment. In circle‘3’, the items 218 are placed automatically in the desired location(e.g., conveyor belt 110) down through a transport means 706 (e.g., liftplatform 120, tube) adjacent to one end of the shelf 104 when a pickoperation 608 is completed, according to one embodiment. The transportmeans 706 may be a bidirectional, up and down capable lift platform 120.

FIG. 3 is a network view 350 illustrating the authentication andmonitoring of a human user 304, according to one embodiment.Particularly, network view 350 shows computing device 302, a human user304, neural network 306, monitoring function 308, validation function310, telepresence application 312, and positioning device 314, accordingto one embodiment.

The computing device 302 may be a data processing system (e.g., asdescribed in FIG. 12) to automatically perform various functions. Thefunctions may include pick and/or place operation of the item ofinventory 206 from the shelf 104 as instructed by the human user 304that is using a positioning device 314 (e.g., positioning arm, datagloves) to move the robotic arm 100 in a particular way. A human user304 may be an authorized entity that uses the telepresence application312 and/or directs the robotic arm 100 to pick and/or place the items218, according to one embodiment.

The neural network 306 may be a system of programs and/or datastructures that approximates the operation of the brain of human user304 and nervous system. The monitoring function 308 may be a set ofinstructions that performs a specific task of supervising the behaviorof the human user 304 that performs the haptic motion 402. In oneembodiment, a human user 304 can also feel the feedback of the remoterobotic arm 100 as it touches the objects (e.g., items 218) at theremote side. The neural network 306 may continuously learn how to betterpick and/or place a particular type of item (e.g., item of inventory206) from the shelf 104. The control program may learn how to controlthe robotic arm 100 to automatically select the items 218 onto thecounting platform 208 without human intervention or human hapticcontrol, according to one embodiment. In one embodiment, the puppeteer(e.g., the human user 304) can “feel” the puppet stick's feedback (e.g.,the vibration or limit of motion as it contacts something solid at theremote end). This feedback may give the puppeteer (e.g., the human user304) the feeling of holding the actual stick live. In one embodiment,this feedback may be critical to the human user 304 being able tocontrol the arm fluidly.

In another embodiment, a minimum latency delay may be preset based onaverage round trip so that the remote latency, even if it's long, isconsistent 100% of the time. For example, according to one embodiment,if the puppeteer is 300 ms behind the remote puppet, then they may beable to learn to react correctly (e.g., think Pac-Man® and how you knowyou have to push the stick to turn a good 250-500 ms before you get tothe turn). In one embodiment, the entire experience feels like a videogame and that the best users may also be skilled at video game eye-handcoordination.

The validation function 310 may be a set of instructions that performs aspecific task of computing the weight of the item of inventory 206 onthe counting platform 208. The validation function 310 may improveerror-proofing and/or productivity in the picking process. Further, thevalidation function 310 may authenticate the human user 304 as anauthorized user of the telepresence application 312 using a processor114 and a memory 116 of a warehouse management server 112, according toone embodiment.

The telepresence application 312 may be software program operating onthe warehouse management server 112 designed to perform a specificfunction as directed by the human user 304. The functions may includeidentifying that the human user 304 has finished the movement of items218 (e.g., item of inventory 206) and/or instructs the human user 304 tothe next order of eCommerce, according to one embodiment.

The positioning device 314 (e.g., positioning arm, data gloves) may be abody suit and/or device that fit's over the user's (e.g., human user304) hand. The positioning device 314 may be controlled by a human user304 remotely, according to one embodiment. The human user 304 may applya sense of touch through vibrations, motion, and/or force that controlsand/or interacts with the robotic arm 100 to perform the pick and/orplace operations. The haptic motion 402 may be replicated by the roboticarm 100 through an Internet network 108. The positioning device 314 mayact as a transmitter and the robotic arm 100 may act as a receiver inthe warehouse system 150, according to one embodiment.

FIG. 3 illustrates a warehouse management server 112 including adatabase 118 coupled with the processor 114 and memory 116, according toone embodiment. The warehouse management server 112 may becommunicatively coupled to the neural network 306 through the Internetnetwork 108. The human user 304 may be remotely coupled with thepositioning device 314. The computing device 302 may be associated withthe human user 304. The telepresence application 312 may operate on thewarehouse management server 112, according to one embodiment.

FIG. 4 is a conceptual view 450 illustrating the mirroring of a hapticmotion 402 of a human user 304 that is using a positioning device 314,according to one embodiment. Particularly, conceptual view 450 shows ahaptic motion 402, telepresence camera 404, and display screen 406,according to one embodiment.

The haptic motion 402 may be a form of interaction that includes themovement of hands and/or other parts of the human (e.g., human user 304)body. The haptic motion 402 may act as an input to the positioningdevice 314 (e.g., positioning arm, data gloves) by which the robotic arm100 may perform the pick and/or place operation, according to oneembodiment.

The computing device 302 may be a programmable electronic devicedesigned to accept data (e.g., instructions given by the human user 304that is using positioning device 314), perform prescribed operations(e.g., placing, validating, and/or moving the item of inventory 206),and display the results (as seen on the display screen 406). Thetelepresence camera 404 may be an optical instrument mounted on themonitor that captures and/or records the haptic motion 402 of the humanuser 304 with a high level of flexibility, according to one embodiment.

The display screen 406 may be a display part of a monitor that helps thehuman user 304 to push, pull, pick, and/or place the intended item ofinventory 206 from the items 218 stored in the storage bin 106 through acamera (e.g., camera for remote human user 212) affixed to the roboticarm 100, according to one embodiment.

FIG. 5 is a user interface view 550 showing the display screen 406 ofthe telepresence application 312, according to one embodiment. Inanother embodiment, the display screen 406 may display the UniversalProduct Code (UPC) of the item of inventory 206. The display screen 406may also show the snapshot taken at a particular instant of time,according to one embodiment.

FIG. 6 is a pick operation view 650, according to one embodiment.Particularly, pick operation view 650 shows step 602, step 604, step606, pick operation 608, image 610, image 612, image 614, and qualityassurance 616, according to one embodiment.

The step 602 may be an action and/or movement (e.g., haptic motion 402)performed by the human user 304 that is using the positioning device 314(e.g., positioning arm, data gloves) prior to the pick and/or placeoperation of the item of inventory 206 from the storage bin 106 and/ortote 214. The step 604 may be an action and/or movement (e.g., hapticmotion 402) performed by the human user 304 that is using thepositioning device 314 (e.g., positioning arm, data gloves) during thepick and/or place operation of the item of inventory 206 from thestorage bin 106 and/or tote 214, according to one embodiment.

The step 606 may be an action and/or movement (e.g., haptic motion 402)performed by the human user 304 that is using the positioning device 314(e.g., positioning arm, data gloves) after the pick and/or placeoperation of the item of inventory 206 from the storage bin 106 and/ortote 214. The pick operation 608 may involve various instants (e.g.,step 602, step 604, step 606) in which the item of inventory 206 may belifted by the robotic arm 100, according to one embodiment.

The image 610 may be a visible impression captured by the telepresencecamera 404 after performing the step 602 by the human user 304 who isoperating the positioning device 314. The image 612 may be a visibleimpression captured by the telepresence camera 404 after performing thestep 604 by the human user 304 who is operating the positioning device314. The image 614 may be a visible impression captured by thetelepresence camera 404 after performing the step 606 by the human user304 who is operating the positioning device 314, according to oneembodiment.

The quality assurance 616 may be a process-centered approach ensuringthat the robotic arm 100, positioning device 314, and/or the human user304 are meeting the specified requirements. The quality assurance 616may also check the random samples of stocks (e.g., items 218) to improvethe work process and/or efficiency of the warehouse system 150. Thesteps (e.g., step 602, step 604, and step 606) may form the basis for ahigh quality training dataset for the warehouse system 150 to build aneural network 306. The training dataset may be used to train the neuralnetwork 306 to identify certain picks and/or moves that robotic arm 100can do itself. The neural network 306 may become facile with certainpicks and/or moves. The neural network 306 may intercept control of therobotic arm 100 from the human user 304 in the telepresence system,according to one embodiment.

FIG. 6 illustrates the database 118 storing image 610, image 612, andimage 614. The pick operation 608 may include step 602, step 604, andstep 606. The telepresence camera 404 may be communicatively coupled tothe pick operation 608 as performed by the human user 304 who isoperating the positioning device 314. The quality assurance 616 may beassociated with the pick operation 608. The telepresence camera 404 mayrecord the step 602 and associate it with the image 610 before the pickoperation 608. The telepresence camera 404 may record the step 604 andassociate it with image 612 during the pick operation 608. Thetelepresence camera 404 may record the step 606 and associate it withthe image 614 after the pick operation 608, according to one embodiment.

FIG. 7 is a partial view showing dual robotic arms 750 performing thepick and/or place operation, according to one embodiment. Particularly,partial view showing dual robotic arms 750 shows robot 702, robotic arms704, and a transport means 706. A robot 702 may be an electro-mechanicalmachine with dual robotic arms 704 carrying out a complex series ofoperations (e.g., pick, place, push, and/or pull operations) controlledby the human user 304. The robotic arms 704 may be dual mechanical armsdesigned to execute pick, move, and/or place operations. The transportmeans 706 may be a carriable medium (e.g., a lift platform 120 and/ortube) adjacent to the one end of the shelf 104 to bring the item ofinventory 206 up and/or down through the shelf 104, according to oneembodiment.

FIG. 8 is a critical path view 850 for automatically placing the items218 in the designated location (e.g., tote 214, storage bin 106) upand/or down through a transport means 706 (e.g., a lift platform 120,tube) adjacent to one end of the shelf 104 when a pick operation 608 iscompleted, according to one embodiment. In operation 802, the human user304 may perform the haptic motion 402. In operation 804, the human user304 may remotely control the set of robotic arms (e.g., robotic arm 100)using the positioning device 314 through an Internet network 108. Inoperation 806, the human user 304 may detect and/or drive control themotion of the robotic arm 100, according to one embodiment. In operation808, the warehouse management server 112 may direct the human user 304of the telepresence application 312 to each location on the shelf 104,according to one embodiment. In operation 810, sensor and motor controlsignals may be sent to the warehouse management server 112 to activateand/or coordinate the desired movement and/or action of the robotic arm100.

In operation 812, the robotic arm 100 may imitate the haptic motion 402of the human user 304 that is using the positioning device 314. Inoperation 814, the robotic arm 100 may place an item of inventory 206 ona counting platform 208 using an end effector 210, according to oneembodiment. In operation 816, the warehouse management server 112 mayvalidate the item of inventory 206 based on the weight of the item(e.g., item of inventory 206) on the counting platform 208. In operation818, the robotic arm 100 may automatically place the items 218 (e.g.,item of inventory 206) in the designated location (e.g., tote 214,storage bin 106) down through a transport means 706 (e.g., lift platform120, tube) adjacent to one end of the shelf 104 when a pick operation iscompleted, according to one embodiment.

FIG. 9 is a process flow 950 illustrating automatic placement of items218 in a desired location (e.g., conveyor belt 110) when the pickoperation 608 is completed, according to one embodiment. In operation902, a robotic arm 100 may be mounted at an end of a row 102 of a shelfof inventory (e.g., shelf 104) on a set of rails 202 affixed to the row102 of a shelf 104. In operation 904, the robotic arm 100 may bepermitted to move horizontally along the three axes along the row 102 ofthe shelf 104. In operation 906, the robotic arm 100 may be repositionedalong the three axes using a set of actuators 204 (e.g., backdrivable,an electrosensing, an electric, a magnetic, a hydraulic, and/orpneumatic actuators).

In operation 908, a haptic motion 402 of a human user 304 may bemirrored contemporaneously that is remotely using a positioning device314 (e.g. positioning arm, data gloves) that is communicatively coupledwith the robotic arm 100 through an Internet network 108. In operation910, the robotic arm 100 may be automatically repositioned along thethree axes using the set of actuators 204 responsive to the hapticmotion 402 of the human user 304 remotely using the positioning device314. In operation 912, an item of inventory 206 may be placed on acounting platform 208 in front of the robotic arm 100 using an endeffector 210 of the robotic arm 100 based on an action of a human (e.g.,haptic motion 402 of the human user 304) that views the item ofinventory 206 on the shelf 104 through a camera (e.g., camera for remotehuman user 212).

The camera (e.g., camera for remote human user 212) may be affixed tothe robotic arm 100 which is communicatively coupled with a computingdevice 302 associated with the human user 304 operating the positioningdevice 314 (e.g. positioning arm, data gloves). In operation 914, theitem of inventory 206 may be moved to a designated location (e.g., tote214, storage bin 106) adjacent to the robotic arm 100 using the endeffector 210. In operation 916, the items 218 (e.g., item of inventory206) may be automatically placed in the designated location (e.g., tote214, storage bin 106) up and/or down through a transport means 706(e.g., lift platform 120, tube) adjacent to one end of the shelf 104when a pick operation 608 is completed, according to one embodiment.

FIG. 10 is a process flow 1050 illustrating repositioning of the roboticarm 100 using a set of actuators 204 based on mirrored haptic motion 402of the human user 304, according to one embodiment. In operation 1002,telepresence application 312 may determine that a remote positioningdevice 314 (e.g., positioning arm, data gloves) associated with a humanuser 304 is communicatively coupled with the telepresence application312. In operation 1004, telepresence application 312 may determine thehuman user 304 as an authorized user of the telepresence application 312using a processor 114 and a memory 116 of a warehouse management server112 on which the telepresence application 312 operates, according to oneembodiment.

In operation 1006, a robotic arm 100 may be automatically allocated in adistribution center that is assigned to the human user 304 based on ane-commerce order. In operation 1008, the robotic arm 100 may becontemporaneously repositioned along the three axes using a set ofactuators 204 based on a mirrored haptic motion 402 of the human user304 that is remotely using a positioning device 314 (e.g., positioningarm, data gloves) that is communicatively coupled with the robotic arm100 through an Internet network 108, according to one embodiment.

FIG. 11 illustrates the various embodiments of the robotic arm 100,according to one embodiment. In ‘1102’, the worker may place the inbounditems 218 on the conveyor belt 110. The robot may be mounted on the setof rails attached across the conveyor belt 110. The robot may push theitem of inventory 206 into the designated container. The container maylight to be picked up. The view ‘1104’ shows various features of therobotic arm 100, the box (e.g., storage bin 106) may be angled back onthe shelf 104 to hold the items 218 and/or prevent the items 218 fromfalling off the shelf 104. The counting platform 208 and robot 702 maytravel along the shelf 104. The camera for remote worker (e.g., camerafor remote human user 212) may provide a close-up view of the items 218inside the storage bin 106 to the human user 304 that is operating thetelepresence application 312. View ‘1106’ shows the top view of theoperation of the robotic arm 100 along the row 102 of the shelf 104.

FIG. 12 is a schematic diagram of specific computing device 1280 ofspecific computing device 1200 that can be used to implement the methodsand systems disclosed herein, according to one embodiment. FIG. 12 is aschematic diagram of specific computing device 1280 of the specificcomputing device 1200 (e.g., computing device 302 of FIG. 3-4) and aspecific mobile computing device 1230 that can be used to perform and/orimplement any of the embodiments disclosed herein.

The specific computing device 1200 may represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and/or otherappropriate computers. The specific mobile computing device 1230 mayrepresent various forms of mobile devices, such as smartphones, cameraphones, personal digital assistants, cellular telephones, and othersimilar mobile devices. The components shown here, their connections,couples, and relationships, and their functions, are meant to beexemplary only, and are not meant to limit the embodiments describedand/or claimed.

The specific computing device 1200 may include a processor 1202 (e.g.,processor 114 of FIGS. 1 and 3), a memory 1204 (e.g., memory 116 ofFIGS. 1 and 3), a storage device 1206, a high speed interface 1208coupled to the memory 1204 (e.g., memory 116 of FIGS. 1 and 3) and aplurality of high speed expansion ports 1210, and a low speed interface1212 coupled to a low speed bus 1214 and a storage device 1206. In oneembodiment, each of the components heretofore may be inter-coupled usingvarious buses, and may be mounted on a common motherboard and/or inother manners as appropriate.

The processor 1202 (e.g., processor 114 of FIGS. 1 and 3) may processinstructions for execution in the specific computing device 1200 (e.g.,computing device 302 of FIG. 3-4), including instructions stored in thememory 1204 (e.g., memory 116 of FIGS. 1 and 3) and/or on the storagedevice 1206 to display a graphical information for a GUI on an externalinput/output device, such as a display unit 1216 coupled to the highspeed interface 1208. In other embodiments, multiple processor(s) 1202(e.g., processor 114 of FIGS. 1 and 3) and/or multiple buses may beused, as appropriate, along with multiple memories and/or types ofmemory 1204 (e.g., memory 116 of FIGS. 1 and 3). Also, a plurality ofspecific computing device 1200 may be coupled with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, and/or a multi-processor system).

The memory 1204 (e.g., memory 116 of FIGS. 1 and 3) may be coupled tothe specific computing device 1200 (e.g., computing device 302 of FIG.3-4). In one embodiment, the memory 1204 (e.g., memory 116 of FIGS. 1and 3) may be a volatile memory. In another embodiment, the memory 1204(e.g., memory 116 of FIGS. 1 and 3) may be a non-volatile memory. Thememory 1204 (e.g., memory 116 of FIGS. 1 and 3) may also be another formof computer-readable medium, such as a magnetic and/or an optical disk.The storage device 1206 may be capable of providing mass storage for thespecific computing device 1200 (e.g., computing device 302 of FIG. 3-4).

In one embodiment, the storage device 1206 may be included of a floppydisk device, a hard disk device, an optical disk device, a tape device,a flash memory and/or other similar solid state memory device. Inanother embodiment, the storage device 1206 may be an array of thedevices in a computer-readable medium previously mentioned heretofore,computer-readable medium, such as, and/or an array of devices, includingdevices in a storage area network and/or other configurations.

A computer program may be included of instructions that, when executed,perform one or more methods, such as those described above. Theinstructions may be stored in the memory 1204 (e.g., memory 116 of FIGS.1 and 3), the storage device 1206, a memory 1204 (e.g., memory 116 ofFIGS. 1 and 3) coupled to the processor 1202 (e.g., processor 114 ofFIGS. 1 and 3), and/or a propagated signal.

The high speed interface 1208 may manage bandwidth-intensive operationsfor the specific computing device 1200 (e.g., computing device 302 ofFIG. 3-4), while the low speed interface 1212 may manage lowerbandwidth-intensive operations. Such allocation of functions isexemplary only. In one embodiment, the high speed interface 1208 may becoupled to the memory 1204 (e.g., memory 116 of FIGS. 1 and 3), thedisplay unit 1216 (e.g., through a graphics processor and/or anaccelerator), and to the plurality of high speed expansion ports 1210,which may accept various expansion cards.

In the embodiment, the low speed interface 1212 may be coupled to thestorage device 1206 and the low speed bus 1214. The low speed bus 1214may be included of a wired and/or wireless communication port (e.g., aUniversal Serial Bus (“USB”), a Bluetooth® port, an Ethernet port,and/or a wireless Ethernet port). The low speed bus 1214 may also becoupled to scan unit 1228, a printer 1226, a keyboard, a mouse 1224, anda networking device (e.g., a switch and/or a router) through a networkadapter.

The specific computing device 1200 (e.g., computing device 302 of FIG.3-4) may be implemented in a number of different forms, as shown in theFIG. 1280. In one embodiment, the specific computing device 1200 may beimplemented as a standard server 1218 (e.g., warehouse management server112) and/or a group of such servers. In another embodiment, the specificcomputing device 1200 (e.g., computing device 302 of FIG. 3-4) may beimplemented as part of a rack server system 1222. In yet anotherembodiment, the specific computing device 1200 (e.g., computing device302 of FIG. 3-4) may be implemented as a general computer 1220 such as alaptop and/or desktop computer. Alternatively, a component from thespecific computing device 1200 may be combined with another component ina specific mobile computing device 1230.

In one or more embodiments, an entire system may be made up of aplurality of specific computing device 1200 and/or a plurality ofspecific computing device 1200 coupled to a plurality of specific mobilecomputing device 1230.

In one embodiment, the specific mobile computing device 1230 may includea mobile compatible processor 1232, a mobile compatible memory 1234, andan input/output device such as a mobile display 1254, a communicationinterface 1246, and a transceiver 1244, among other components. Thespecific mobile computing device 1230 may also be provided with astorage device, such as a Microdrive and/or other device, to provideadditional storage. In one embodiment, the components indicatedheretofore are inter-coupled using various buses, and several of thecomponents may be mounted on a common motherboard.

The mobile compatible processor 1232 may execute instructions in thespecific mobile computing device 1230, including instructions stored inthe mobile compatible memory 1234. The mobile compatible processor 1232may be implemented as a chipset of chips that include separate andmultiple analog and digital processors. The mobile compatible processor1232 may provide, for example, for coordination of the other componentsof the specific mobile computing device 1230, such as control of user(e.g., human user 304 of FIG. 3-4) interfaces, applications run by thespecific mobile computing device 1230, and wireless communication by thespecific mobile computing device 1230.

The mobile compatible processor 1232 may communicate with a user (e.g.,human user 304 of FIG. 3-4) through the control interface 1236 and thedisplay interface 1256 coupled to a mobile display 1254. In oneembodiment, the mobile display 1254 may be a Thin-Film-Transistor LiquidCrystal Display (“TFT LCD”), an Organic Light Emitting Diode (“OLED”)display, and another appropriate display technology. The displayinterface 1256 may include appropriate circuitry for driving the mobiledisplay 1254 to present graphical and other information to a user (e.g.,human user 304 of FIG. 3-4).

The control interface 1236 may receive commands from a user (e.g., humanuser 304 of FIG. 3-4) and convert them for submission to the mobilecompatible processor 1232. In addition, an external interface 1238 maybe provided in communication with the mobile compatible processor 1232,so as to enable near area communication of the specific mobile computingdevice 1230 with other devices. External interface 1238 may provide, forexample, for wired communication in some embodiments, and/or forwireless communication in other embodiments, and multiple interfaces mayalso be used.

The mobile compatible memory 1234 may be coupled to the specific mobilecomputing device 1230. The mobile compatible memory 1234 may beimplemented as a volatile memory and a non-volatile memory. Theexpansion memory 1240 may also be coupled to the specific mobilecomputing device 1230 through the expansion interface 1242, which mayinclude, for example, a Single In Line Memory Module (“SIMM”) cardinterface. The expansion memory 1240 may provide extra storage space forthe specific mobile computing device 1230, and/or may also store anapplication and/or other information for the specific mobile computingdevice 1230.

Specifically, the expansion memory 1240 may include instructions tocarry out the processes described above. The expansion memory 1240 mayalso include secure information. For example, the expansion memory 1240may be provided as a security module for the specific mobile computingdevice 1230, and may be programmed with instructions that permit secureuse of the specific mobile computing device 1230. In addition, a secureapplication may be provided on the SIMM card, along with additionalinformation, such as placing identifying information on the SIMM card ina non-hackable manner.

The mobile compatible memory 1234 may include a volatile memory (e.g., aflash memory) and a non-volatile memory (e.g., a non-volatilerandom-access memory (“NVRAM”)). In one embodiment, a computer programincludes a set of instructions that, when executed, perform one or moremethods. The set of instructions may be stored on the mobile compatiblememory 1234, the expansion memory 1240, a memory coupled to the mobilecompatible processor 1232, and a propagated signal that may be received,for example, over the transceiver 1244 and/or the external interface1238.

The specific mobile computing device 1230 may communicate wirelesslythrough the communication interface 1246, which may be included of adigital signal processing circuitry. The communication interface 1246may provide for communications using various modes and/or protocols,such as: a Global System for Mobile Communications (“GSM”) protocol, aShort Message Service (“SMS”) protocol, an Enhanced Messaging System(“EMS”) protocol, a Multimedia Messaging Service (“MMS”) protocol, aCode Division Multiple Access (“CDMA”) protocol, Time Division MultipleAccess (“TDMA”) protocol, a Personal Digital Cellular (“PDC”) protocol,a Wideband Code Division Multiple Access (“WCDMA”) protocol, a CDMA2000protocol, and a General Packet Radio Service (“GPRS”) protocol.

Such communication may occur, for example, through the transceiver 1244(e.g., radio-frequency transceiver). In addition, short-rangecommunication may occur, such as using a Bluetooth®, Wi-Fi, and/or othersuch transceiver. In addition, a GPS (“Global Positioning System”)receiver module 1258 may provide additional navigation-related andlocation-related wireless data to the specific mobile computing device1230, which may be used as appropriate by a software application runningon the specific mobile computing device 1230.

The specific mobile computing device 1230 may also communicate audiblyusing an audio codec 1248, which may receive spoken information from auser (e.g., human user 304 of FIG. 3-4) and convert it to usable digitalinformation. The audio codec 1248 may likewise generate audible soundfor a user (e.g., human user 304 of FIG. 3-4), such as through a speaker(e.g., in a handset of the specific mobile computing device 1230). Sucha sound may include a sound from a voice telephone call, a recordedsound (e.g., a voice message, a music files, etc.) and may also includea sound generated by an application operating on the specific mobilecomputing device 1230.

The specific mobile computing device 1230 may be implemented in a numberof different forms, as shown in the FIG. 1280. In one embodiment, thespecific mobile computing device 1230 may be implemented as a smartphone1250. In another embodiment, the specific mobile computing device 1230may be implemented as a personal digital assistant (“PDA”). In yetanother embodiment, the specific mobile computing device 1230 may beimplemented as a tablet device 1252.

It will be appreciated that the various embodiments described herein mayoperate in additional embodiments. For example, the robotic arm 100 onthe shelf 104 may operate in context of a vending machine rather than ina warehouse system 150 (e.g., a giant urban vending machine to dispensea wide array of goods and services directly to customers placing ordersthrough the Internet using a mobile device or phone). The vendingmachine in this alternate embodiment may operate through the Internet toautomatically pick and/or place items inside of a dispenser of productsfor sale directly to consumers. Any of the various embodiments describedin FIGS. 1-12 may operate in this vending machine context (e.g., thevending machine may be connected to a telepresence application throughan Internet network).

Further, the various embodiments of FIGS. 1-12 may also operate with agravity fed tote chute system (e.g., bound totes may be dropped into thetop of the chute and picked totes drop out the bottom) rather than atube as described in the various embodiments. At the robotic arm levelthe chute may have a gap where users can place a picked tote (to fallthrough to the bottom) and/or an actuator that stacks, top-loaded to putaway totes. The actuator may stop the stack from entering the slot untilthe robotic arm is ready to grab that tote for put away (e.g., similarlyto a Pez Dispenser® that goes in both directions with the opening in themiddle). Further, to improve efficiency, items may need to be put intototes before they are loaded into the internet vending machine. In onealternate embodiment, the vending machine described herein that mayoperate with any of the various embodiments of FIGS. 1-12 mayinteroperate with standard inbound distribution center logistics for asystem to function.

A “totifying” system may be utilized independently or in conjunctionwith the various embodiments described herein to deploy the totingsystem. For example, a conveyor with individual controllable rollers(e.g., having a precision dc motor) may be utilized. This conveyor mayroll in front of 50 tote positions along one side of the conveyor. Onthe other side may be 5-10 robotic arms that gantry side-to-side justlike they do on the shelves. At the end of the conveyor, there may be1-2 people pulling random objects out of inbound shipping cartons. Everytime a worker throws an item onto the conveyor, it may roll away to a“scan zone” where it's identified. Every identified item type may have apreassigned bin along the conveyor. These may be scheduled based on whatwe expect in both the inbound carton and/or total inbound shipment. Theconveyor may roll that item to a spot in front of the item's assignedtote. A robotic arm may roll up to that position. A human may beattached to the robotic arm's telepresence and may validate the item andmay then push the item into the tote. When a tote is completed (per theexpected shipment), it may light up. A human may put it on a conveyorthat may take it to the loading station at the top of the vendingmachine. This “totifying system” may be perfect for anysort/inspect/select conveyor system such as those found in agriculture,confectionery operations, etc.

In other alternate embodiments, additional concepts are contemplatedincluding * A system of cubbies of various sizes that may be availableto approved tenants to store items for shipment. * A tenant API that mayallow vendors to reserve cubbies of a certain size starting on a certaindate. * A “public” vendor API that may enable: (1) An API key managementsystem to optionally permit tenants to restrict API use per item to aspecified set of users. (e.g., approved web retailers). (2) Permitanyone (or authorized users) with an appropriately validated API key toquery item availability. (3) Permit anyone (or authorized users) with anappropriately validated API key to query item cost. (4) Permit anyone(or authorized users) with an appropriately validated API key to selectdelivery method and cost. (5) Permit anyone (or authorized users) withan appropriately validated API key to order items out of the machineafter settlement. (6) Track location of shipped items. (7) A “private”vendor API may: allow the tenant (only) to track and ship items from acubby. (8) An economic equilibrium formula may be used in conjunctionwith any of the embodiment disclosed herein including (9) The “holdingcost” of an item that may not move/sell quickly may go from 0->N $/day.(This starts from the cubby reserve date.) (10) The ability to(optionally) accept bid prices for items in the machine. (11) Theability to optionally present liquidation/clearance via the public API.(12) The ability to optionally transfer “ownership” of a cubby (with itsitems) to a different tenant. (13) A wholesale liquidation option mayalso be provided. (14) The ability to store the same item in multiplecubbies is also contemplated in one embodiment. (15) The ability toconsolidate the same item from multiple cubbies down to one cubby, isyet another contemplated embodiment consistently with any of theembodiments described herein. In one alternate embodiment, “specialtystorefronts” may be created near the vending machine that allow localvendors to store and retrieve items from the vending machine, so thatitems can be accessed by the public directly.

In another alternate embodiment, since the various embodiments describedin FIGS. 1-12 may have captured video of every pick, this video and/oraudio information may be relayed to several downstream and/or historicalprocesses such as a QA pick validation team. For example, through thismethodology: * every pick may be reviewable by a statistical QA processvia telepresence, according to one embodiment. * every pick may have avendor review record for customer issue validation, according to oneembodiment. * every pick may have the chance to show the customer theiritem getting picked for their own personal review, according to oneembodiment. * every pick may allow supervisors to grade trainees,according to one embodiment. * every pick may allow supervisors to trainnew employees, according to one embodiment. * every pick may allowquality assurance, managers, independent auditors, and/or independentcontractors to review packaging and process on behalf of the customer,according to one embodiment. * loss prevention review may be possible,according to one embodiment. * schedule 1 and 2 pharmaceutical controlfunction may be possible, according to one embodiment. In other words,the video record may have significant downstream, review and audit valuewhen utilized in conjunction with the various embodiments of FIGS. 1-12.

An example embodiment will now be described. A Delta DistributionCenter, Inc. based in North America may be a manufacturer anddistributor of specialty fans for the high-end consumer market. A DeltaDistribution Center, Inc. may sell its goods directly to the customerswho have ordered products through catalogues or online stores. The DeltaDistribution Center, Inc. may have a warehouse in Tampa Fla. to serve asa single location to stock and fulfill a vast number of products. Bydeploying the various embodiments described in FIGS. 1-12, DeltaDistribution Center, Inc. may have now have sufficient space to storestock in many different packaging styles and/or quantities. Further, themanagement team of Delta Distribution Center, Inc. may not need to hireadditional warehouse operators to pick and/or place items when salesincrease in the holiday season by deploying the various embodiments ofFIGS. 1-12.

In addition, material handling may no longer be a primary source ofhazard in the Delta Distribution warehouse in Tampa Fla. For example, awarehouse operator John Smith may no longer need to enter the liftbasket and manually perform hazardous tasks such as picking andcollecting items from higher shelves, thanks to the various embodimentsdescribed in FIGS. 1-12. This may reduce labor expenses in training andskilled labor, making Delta Distribution a more profitable business.Also, the warehouse operator of the Delta Distribution warehouse inTampa may no longer need to climb ladders to get to top shelves or racksto access needed inventory, thanks to the various embodiments describedin FIGS. 1-12. Therefore, Delta Distribution may save money by reducingcosts for professional hazard insurances for workplace injury and/oroccupational risks, thanks to the various embodiments described in FIGS.1-12.

In the competitive high-end fan industry, the speed of fulfilling ordersfrom the time an order is placed online to when it is shipped and sentto a customer may be important. For this reason, the management of DeltaDistribution Center, Inc. may have adopted a warehouse system (asdescribed in the various embodiments of FIGS. 1-12) that enables them toautomate processes and/or improve operations to save time and/or money.

For example, thanks to the various embodiments of the FIGS. 1-12, theDelta Distribution Center, Inc. may implement the methods and/or systemdescribed herein to perform pick and/or place operations through roboticarms based on a telepresence application. Instead of having warehouseoperators manually study paper maps and then go to the shelf and pickintended items of inventory from rows of a shelf, the variousembodiments described herein may facilitate hands-free order picking andeliminate extra labor.

The management of Delta Distribution Center, Inc. may now integrate newtechniques such as mobile robotics and telepresence technology intomaterial handling to ensure quality and/or accuracy using theembodiments of FIGS. 1-12. A robust warehouse management system (asdescribed in the various Figures from FIGS. 1-12) may be employed in theDelta Distribution Center, Inc. for its efficient working. The warehouseoperator (e.g., human user 304) of the Delta Distribution Center, Inc.may handle the task of picking and/or placing the item of inventory justby sitting at one place in the control room of the distribution center.

For example, the warehouse operator (e.g., human user 304) may wearpositioning arm (e.g., positioning device 314) to perform the motion(e.g., haptic motion 402) that may be mimicked by the robotic arm (e.g.,robotic arm 100) as mounted on the rails (e.g., set of rails 202) acrosseach row of the shelf (e.g., row 102 of the shelf 104) throughtelepresence technology (e.g., telepresence application 312). Thewarehouse operator (e.g., human user 304) may have a sense of being onlocation (e.g., near to the shelf 104) by employing the telepresencetechnology in the control room of the Delta Distribution Center, Inc.(and/or at an offsite location). The warehouse operator (e.g., humanuser 304) may accurately recognize the position and/or orientation of anitem (e.g., item of inventory 206) through the camera (e.g., camera 220)mounted on the top of the row (e.g., row 102) just above the storage bin106.

By employing the methods and/or systems described in the FIGS. 1-12, theDelta Distribution Center, Inc. may reduce processing time, enhanceproductivity, reduce labor and operational costs making theaffordability enticing, minimize travel time, increase order accuracyand/or number of orders that can be picked each day, have fewer safetyincidents, and/or improve cycle times. As a result, the pick and/orplace operations in the Delta Distribution Center, Inc. may now beaccelerated, making the company thrive in an otherwise competitivemarket.

It should be noted that a variation of the various embodiments describedherein could include mounting the robotic arm onto a bar or beam abovean item of inventory—this variation could apply to situations where theitem of inventory is on the floor of the facility such as boxes or casesstored on pallets, or the item of inventory is hanging on a rod such asgarments on hangers.

Various embodiments of the systems and techniques described here can berealized in at least one of a digital electronic circuitry, anintegrated circuitry, a specially designed application specificintegrated circuits (“ASICs”), a piece of computer hardware, a firmware,a software application, and a combination thereof. These variousembodiments can include embodiment in one or more computer programs thatare executable and/or interpretable on a programmable system includingat least one programmable processor, which may be special or generalpurpose, coupled to receive data and instructions from, and to transmitdata and instructions to, a storage system, at least one input device,and at least one output device.

These computer programs (also known as programs, software, softwareapplications, and/or code) comprise machine-readable instructions for aprogrammable processor, and can be implemented in a high-levelprocedural and/or object-oriented programming language, and/or inassembly/machine language. As used herein, the terms “machine-readablemedium” and/or “computer-readable medium” refers to any computer programproduct, apparatus and/or device (e.g., magnetic discs, optical disks,memory, and/or Programmable Logic Devices (“PLDs”)) used to providemachine instructions and/or data to a programmable processor, includinga machine-readable medium that receives machine instructions as amachine-readable signal. The term “machine-readable signal” refers toany signal used to provide machine instructions and/or data to aprogrammable processor.

To provide for interaction with a human user, the systems and techniquesdescribed here may be implemented on a computing device having a displaydevice (e.g., a cathode ray tube (“CRT”) and/or liquid crystal display(“LCD”) monitor) for displaying information to the human user, keyboardand a mouse by which the human user can provide input to the computer(e.g., computing device). Other kinds of devices can be used to providefor interaction with a human user as well; for example, feedbackprovided to the human user can be any form of sensory feedback (e.g.,visual feedback, auditory feedback, and/or tactile feedback) and inputfrom the human user can be received in any form, including acoustic,speech, and/or tactile input.

The systems and techniques described here may be implemented in acomputing device that comprises at least one of a back end component(e.g., as a data server), a middleware component (e.g., an applicationserver), a front end component (e.g., a client computer having agraphical user interface, and/or a Web browser through which a humanuser can interact with an embodiment of the systems and techniquesdescribed here), and a combination thereof. The components of the systemmay also be coupled through a communication network.

The communication network may comprise at least one of a local areanetwork (“LAN”) and a wide area network (“WAN”) (e.g., the Internet).The warehouse system 150 can comprise at least one of a client and aserver. In one embodiment, the client and the server are remote fromeach other and interact through the communication network.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of the claimed invention. In addition, the logicflows depicted in the figures do not require the particular order shown,or sequential order, to achieve desirable results. In addition, othersteps may be provided, or steps may be eliminated, from the describedflows, and other components may be added to, or removed from, thedescribed systems. Accordingly, other embodiments are within the scopeof the following claims.

It may be appreciated that the various systems, methods, and apparatusdisclosed herein may be embodied in a machine-readable medium and/or amachine accessible medium compatible with a data processing system(e.g., a computer system), and/or may be performed in any order.

The structures and modules in the figures may be shown as distinct andcommunicating with only a few specific structures and not others. Thestructures may be merged with each other, may perform overlappingfunctions, and may communicate with other structures not shown to beconnected in the figures. Accordingly, the specification and/or drawingsmay be regarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: mounting a robotic arm at anend of a row of a shelf of inventory on a set of rails affixed to therow of a shelf; permitting the robotic arm to move horizontally alongthe three axes along the row of the shelf; repositioning the robotic armalong any one of the three axes using a set of actuators, wherein eachof the set of actuators are any of a backdrivable, an electrosensing, anelectric, a magnetic, a hydraulic, and pneumatic actuators;contemporaneously mirroring a haptic motion of a human user that isremotely using a positioning device that is communicatively coupled withthe robotic arm through an Internet network; automatic repositioning ofthe robotic arm along any one of the three axes using the set ofactuators responsive to the haptic motion of the human user remotelyusing the positioning device; placing an item of inventory on a countingplatform in front of the robotic arm using an end effector of therobotic arm based on an action of a human that views the item ofinventory on the shelf through a camera affixed to the robotic arm whichis communicatively coupled with a computing device associated with thehuman user operating the positioning device; moving the item ofinventory to a designated location adjacent to the robotic arm using theend effector; and automatically placing the items in the designatedlocation down through a transport means adjacent to one end of the shelfwhen a pick operation is completed, wherein the designated location isat least one of a tote and a storage bin, and wherein the transportmeans is at least one of a tube and a lift platform to bring the totefrom the designated location to a desired location of the shelf.
 2. Themethod of claim 1, wherein the item of inventory is placed on a platformhaving an angled surface such that the storage bin in which the item ofinventory is placed is angled upward and does not fall off the shelfwhen placed on the counting platform, and wherein the item of inventoryis validated based on a weight of an item on the counting platform. 3.The method of claim 1, wherein a telepresence application is used toprovide the positioning device control over the robotic arm remotelythrough the Internet network.
 4. The method of claim 3, wherein theshelf includes a plurality of rows of the shelf and a plurality ofrobotic arms, wherein each of the rows of the shelf includes one or morerobotic arms that horizontally traverses a particular row in which it isaffixed.
 5. The method of claim 4, wherein each of the robotic armscoordinates with a warehouse management server to automatically directthe human user of the telepresence application to each location on theshelf associated with the items needed to be fulfilled in an e-commerceorder.
 6. The method of claim 4, wherein each item of an e-commerceorder is automatically deposited through respective ones of theplurality of robotic arms into a packing box on a conveyer belt of adistribution center when the items are deposited down through the tubeadjacent to the one end of the shelf.
 7. The method of claim 6, whereinthe telepresence application detects that the human has completed amovement of the items onto the counting platform and directs the humanuser to a next one of a plurality of the robotic arms positioned infront of the storage bin where a selection is needed.
 8. The method ofclaim 7, wherein a neural network automatically monitors a behavior ofthe human user and continuously learns how to improve the pick and placeof a particular type of item from the shelf such that, over time, acontrol program learns how to control the robotic arm to automaticallyselect the items onto the counting platform without a human interventionor a human haptic control.
 9. The method of claim 1, wherein the endeffector of the robotic arm is at least one of a supple rubber endpoint, a gripping arm, a sticky polymer end, an impactive end effector,an ingressive end effector, an astrictive end effector, and acontigutive end effector.
 10. The method of claim 1, wherein a robot hastwo robotic arms with actuator control to permit the human user to graband lift the items.
 11. The method of claim 1, wherein the robotic armcan safely operate around the human operators in a distribution centerby slowly coming to a stop and not having to stop abruptly when anadjacent object is detected which is likely to obstruct a motion of therobotic arm through compliance to prevent users and operators beinginjured.
 12. A method of a telepresence application comprising:determining that a remote positioning device associated with a humanuser is communicatively coupled with the telepresence application;validating the human user as an authorized user of the telepresenceapplication using a processor and a memory of a warehouse managementserver on which the telepresence application operates; automaticallyallocating a robotic arm in a distribution center that is assigned tothe human user based on an e-commerce order; contemporaneouslyrepositioning the robotic arm along any one of the three axes using aset of actuators based on a mirrored haptic motion of the human userthat is remotely using a positioning device that is communicativelycoupled with the robotic arm through an Internet network.
 13. The methodof the telepresence application of claim 12 further comprising:providing a view of an item of inventory directly in front of therobotic arm to the human user who is remotely controlling the roboticarm through the positioning device, wherein the robotic arm to place theitem of inventory on a counting platform in front of the robotic armusing an end effector of the robotic arm based on an action of a humanthat views the item of inventory on a shelf through a camera affixed tothe robotic arm which is communicatively coupled with a computing deviceassociated with the human user operating the positioning device based ona place command from the telepresence application; wherein the roboticarm to validate the item of inventory based on a weight of the item onthe counting platform based on a validate command from the telepresenceapplication; wherein the robotic arm to move the item of inventory in alocation adjacent to the robotic arm using the end effector based on amove command from the telepresence application; and moving the items atthe location down through a tube adjacent to one end of the shelf when apick operation is completed based on a complete command from thetelepresence application.
 14. The method of claim 12, wherein an item ofinventory is placed on a platform having an angled surface such that astorage bin in which the item of inventory is placed is angled upwardand does not fall off a shelf when placed on a counting platform. 15.The method of claim 12, wherein the telepresence application is used toprovide the positioning device control over the robotic arm remotelythrough the Internet network.
 16. The method of claim 14: wherein theshelf includes a plurality of rows of the shelf and a plurality ofrobotic arms, wherein each of the rows of the shelf includes a differentrobotic arm, and wherein each of the rows of the shelf includes one ormore different robotic arms that horizontally traverses a particular rowin which it is affixed.
 17. The method of claim 15, wherein each of therobotic arms to coordinate with the warehouse management server toautomatically direct the human user of the telepresence application toeach location on a shelf associated with the items needed to befulfilled in the e-commerce order.
 18. The method of claim 16, whereineach item of the e-commerce order is automatically deposited throughrespective ones the plurality of robotic arms into a packing box on aconveyer belt of the distribution center when the items are depositeddown through a tube adjacent to one end of the shelf.
 19. The method ofclaim 17, wherein the telepresence application detects that a human hascompleted a movement of the items onto a counting platform and directsthe human user to a next one of a plurality of the robotic armspositioned in front of a storage bin where a selection is needed. 20.The method of claim 18, wherein a neural network to automaticallymonitor a behavior of the human user and to continuously learn how tobetter pick and place a particular type of item from the shelf suchthat, over time, the robotic arm is able to automatically pick the itemand place the item at a location without requiring mirroring of a hapticmotion of the human user that is remotely using a positioning arm. 21.The method of claim 12: wherein an end effector of the robotic arm is atleast one of a supple rubber end point, a gripping arm, a sticky polymerend, an impactive end effector, an ingressive end effector, anastrictive end effector, and a contigutive end effector, and wherein therobotic arm to safely operate around the human operators in thedistribution center by slowly coming to a stop and not having to stopabruptly when an adjacent object is detected which is likely to obstructa motion of the robotic arm.
 22. A warehouse system, comprising: a setof robotic arms that are each affixed to the individual rows of shelvingin a distribution center to reposition horizontally along the individualrows of shelving along any one of the three axes using a set ofactuators; an Internet network; a warehouse management server having atelepresence application coupled with the set of robotic arms throughthe Internet network to provide a human user remote control over each ofa set of the robotic arms through the Internet network; a positioningdevice coupled with each of the set of robotic arms and the warehousemanagement server having the telepresence application through theInternet network that is haptically controlled by a human user in amanner such that a haptic motion of the human user of the positioningdevice is imitated through a mirrored repositioning of the set ofrobotic arms.
 23. The warehouse system of claim 22, wherein an item ofinventory is placed on a platform having an angled surface such that astorage bin in which the item of inventory is placed is angled upwardand does not fall off a shelf when placed on a counting platform. 24.The warehouse system of claim 22, wherein the telepresence applicationis used to provide the positioning device control over a robotic armremotely through the Internet network.
 25. The warehouse system of claim24: wherein a shelf includes a plurality of rows of the shelf and aplurality of robotic arms, wherein each of the rows of the shelfincludes a different robotic arm, and wherein each of the rows of theshelf includes one or more different robotic arm that horizontallytraverses a particular row in which it is affixed.
 26. The warehousesystem of claim 25, wherein each of the robotic arms to coordinate withthe warehouse management server to automatically direct the human userof the telepresence application to each location on the shelf associatedwith the items needed to be fulfilled in an e-commerce order.
 27. Thewarehouse system of claim 26, wherein each item of the e-commerce orderis automatically deposited through respective ones of the plurality ofrobotic arms into a packing box on a conveyer belt of the distributioncenter when the items are deposited down through a tube adjacent to oneend of the shelf.
 28. The warehouse system of claim 27, wherein thetelepresence application detects that a human has completed a movementof the items onto a counting platform and directs the human user to anext one of a plurality of the robotic arms positioned in front of astorage bin where a selection is needed.
 29. The warehouse system ofclaim 28, wherein a neural network to automatically monitor a behaviorof the human user and to continuously learn how to better pick and placea particular type of item from the shelf such that, over time, therobotic arm is able to automatically pick the item and place the item atthe location without requiring mirroring of the haptic motion of thehuman user that is remotely using a positioning arm.
 30. The warehousesystem of claim 22, wherein an end effector of a robotic arm is at leastone of a supple rubber end point, a gripping arm, a sticky polymer end,an impactive end effector, an ingressive end effector, an astrictive endeffector, and a contigutive end effector, and wherein the robotic arm tosafely operate around the human operators in the distribution center byslowly coming to a stop and not having to stop abruptly when an adjacentobject is detected which is likely to obstruct a motion of the roboticarm.